The invention relates to a method and an apparatus for processing 3D image data. In particular, the invention relates to methods and systems in which objects are described in three-dimensional (3D) representation with so-called 3D image data.
By way of example, motor vehicles include microprocessor-controlled systems on which applications are executed that generate 3D image data. To this end, each application builds a separate so-called scene model that describes a 3D scene. In order to represent the 3D scene on a display unit, so-called renderers are used. Said systems can likewise be executed on a microprocessor, in particular on a computer. They substantially serve for processing the 3D image data of the 3D scene such that they are suitable for representation on the display unit.
During a rendering process, for example, an image can be computed from a 3D scene. During the conversion of 3D image data which describe a vector graphic, it is possible in the course of the rendering process for example to convert the vector graphic of an object into an image-point-wise pixel representation of the object in a two-dimensional (2D) computer graphic.
A 3D renderer can, for example, generate a separate 2D graphic from each individual 3D scene. By way of a control component, a so-called layer manager, it is possible to produce a total image for the display on a display unit by superposing different 2D graphics. Here, the individual 2D images are placed one above the other according to a fixed sequence in the form of planes. Contents from a higher plane can here cover contents from a lower plane. For the topmost plane, the visibility of its contents can be ensured.
Owing to such an architecture or data processing based on planes, 3D contents of different applications can be displayed on a common display. It is also possible to ensure here that contents of a safety-relevant application are represented on the display, that is to say that they are not covered by contents of other, non-safety-relevant applications.
FIG. 1 illustrates a corresponding image processing architecture. Here, three user programs run in a plurality of controllers or devices in a motor vehicle, that is to say a user program 1 for pedestrian detection in an onboard controller, a media player 2 in a multimedia system installed in the motor vehicle, and an e-mail application 3 in a smartphone that is temporarily connected in the vehicle. The application programs in each case generate image data that describe a 3D scene 4, 5 or 6. By way of example, it is possible to represent in the scene 4 the outline of a pedestrian, a photo produced by the media player in the scene 5, and an e-mail list in the scene 6. The scene image data are converted into 2D images in renderers 7, 8, 9 associated with the respective application programs. Here, planes are also associated with the application programs 1, 2, 3, wherein the image data 10 are associated with a topmost plane, the image data 11 with a central plane, and the image data 12 with a lower-most plane. A layer-manager application 13 receives the respective image data and superposes them plane-by-plane, with the topmost plane being superposed on all further underlying planes. This produces a 2D image 14, which is output by the layer manager 13.
A disadvantage of the described method of rendering each 3D scene separately and of superposition in a plane-wise manner at a later point is that no space-based coverages or optical effects such as reflections and shadows between elements of different 3D scenes can be computed. In addition, it is occasionally necessary for example in motor vehicles to display specific graphics contents spread over a plurality of display units, for example a first graphics portion in a main display unit in the region of the steering wheel, a second graphics portion in an infotainment system in the center console region, and a third graphics portion in a further infotainment system in the region of the rear seats. Individual graphics portions can here also be displayed in a plurality of display units at the same time. Such requirements are not sufficiently well met by the described plane-based approach.
U.S. Pat. No. 7,145,562 B2 discloses a system for representing two-dimensional and three-dimensional objects in a two-dimensional computer graphic, in which 2D objects and 3D objects can be integrated hierarchically in a common scene. The contents of this publication are hereby incorporated in the present description by reference.
When representing computer graphics, the generation and presentation of the finished image frequently take place in the same device. So as to enable an exchange of graphics contents beyond device limits, the graphics data are generally transmitted in predefined formats, so-called video streams. The video streams are here transmitted to a remote device via an interface and integrated here into the local scene. A disadvantage here is that determining interactions between remote and local scenes and realizing them in the target graphic is possible in most cases only with a high degree of complexity. For example, if objects that cast a shadow, which shadow is intended to be visible also on the objects in the local scene, are present in the transmitted video stream, it is necessary to exchange additional information between the two devices. It also requires a high degree of computing complexity during the production of the target graphic.
It is an object of the invention to allow superposition of images based on 3D input image data with a high degree of exactness and acceptable computing complexity.
This and other objects are achieved in accordance with emdoiments of the invention.
According to the invention, provision is made, for the purpose of processing 3D input image data for displaying images on at least one display unit, for the 3D input image data to be provided scene-by-scene in each case in accordance with an associated 3D image scene. The 3D input image data are each fed to a scene processing unit for management and processing and are processed scene-by-scene in the scene processing unit. Here, at least two 3D image scenes are superposed at least partially and a whole 3D scene is formed and managed. A plurality of 3D output scenes are derived from the whole 3D scene, in which 3D output scenes the superposition takes place in each case in accordance with a perspective viewing location and wherein 3D output image data are produced. The 3D output image data are fed to at least one rendering unit, which is associated with the display unit, for producing target image data that are adapted to the display unit.
The 3D input image data typically contain values for representing 3D objects, in particular 3D coordinate data and object attributes. A 3D scene can, in data-technological terms, be structured in particular via a scene tree or a scene graph. The 3D input image data can be made available in particular by a plurality of client-computer systems. The client-computer systems, the scene processing unit and the rendering unit can be microprocessor-controlled devices and/or be computer program applications loaded and running on such devices.
The invention is based on the finding that the forming of a whole 3D scene makes possible simplified and at the same time exact processing for deriving the 3D output image data from the different 3D input image data or from the 3D image scenes associated therewith. The invention is also based on the finding that the superposition of the 3D image scenes is advantageously possible from the whole 3D scene. Additionally it has been found that it is expedient to include a respective perspective viewing location in the formation of the 3D output image data when superposing from the whole 3D scene.
The perspective viewing location can be included as a camera parameter in the superposition, wherein possible camera parameters to be used are typical perspective parameters such as opening angle of the image scene or distance from the image scene. Camera parameters which can be provided by way of example are:                a transformation matrix which describes the position and orientation of the camera in 3D space and describes the example position and/or relative viewing direction or point at which the camera is directed. In addition, a vector may be provided that indicates a preferential direction, for example the normal.        opening angle in particular in the vertical direction        aspect ratio of the target image, for example 16:9        a so-called near-plane parameter which corresponds to the distance of the camera position from a projection surface, with all elements located in front of said surface not being visible in the target image.        a so-called far-plane parameter which corresponds to the distance of the camera position from a surface that limits the camera space, with all elements located behind said surface not being visible in the target image.        
If near-plane and far-plane parameters are used, a camera only records scene elements that are located between the near plane and the far plane. The surfaces of the planes result from the vertical opening angle, the aspect ratio and the respective value of the near plane or far plane.
The perspective viewing location or camera parameter can in each case correspond to the spatial position, alignment and/or other parameters of the at least one display unit. The display unit can be suitable for representing 3D images. It can also be configured for representing 2D images. Depending on the type of representation, the rendering unit converts the 3D output image data into 3D target image data or into 2D target image data. In the case of 2D target image data, the rendering unit is in particular a raster processor. A plurality of display units may be provided. Different perspective viewing locations or different camera parameters can be associated with different display units. In that case, for deriving the 3D output scenes from the whole 3D scene, the 3D image scenes can be superposed in each case according to the various perspective viewing locations or parameters. It is also possible for a plurality of rendering units to be provided, with which in each case one or a plurality of display units can be associated.
Owing to the globally managed whole 3D scene, the transmission of video streams can be omitted and graphics contents can be exchanged directly between different display units. The incorporation of additional display units is significantly simplified. It may suffice here to adapt in each case a small control portion in the respective system units, for example a small software portion, in order to include a new display unit in the display group consisting of a plurality of display units. It is additionally possible for any devices to contribute new graphics contents to the total scene. A display unit can in each case have a dedicated rendering unit that is physically fixedly connected therewith.
In each case one of at least two priority stages can be associated with the 3D input image data, with a priority stage in each case indicating to what degree the associated 3D image scene remains visible when superposed with another 3D image scene. The 3D output scenes can then be configured such that, when the target image data are produced in the rendering unit, those target image data that are derived from a 3D image scene of a higher priority stage are superposed at a target image coordinate on target image data that are derived from a lower priority stage. It thus becomes possible, for example, for safety-relevant image elements to be superposed on the display unit on other image elements that are not safety-relevant, i.e. to be displayed with a high degree of safety. By way of example, it is possible in a motor vehicle to ensure that the safety-relevant display relating to the level of a selected gear is superposed on the display unit on a non-safety-relevant display of navigation data. The association of the priority stages can take place 3D image scene for 3D image scene, in the scene tree and/or in the form of an attribute. It is additionally possible for provision to be made for, when image data which are characterized as safety-relevant are stored, in particular in the scene management unit and/or in the rendering unit, a memory write protection to take place, as a result of which unintended overwrite of the memory is prevented.
It is possible by way of the invention to process safety-relevant and non-safety-relevant 3D contents of different client-computer systems together in the scene processing unit and to produce therefrom a whole 3D scene with mixed contents. Here, on the one hand, safety-relevant 3D contents can be protected against unintended manipulations and against being covered by using the associated attributes. At the same time it is thereby possible to form a coherent whole 3D scene while taking into consideration coverages of 3D image scenes, optical effects and interactions between the 3D image scenes. It is thus possible with the invention to indicate a so-called mixed criticality system for image processing in particular in a motor vehicle.
In order to ensure that target image data relating to safety-relevant displays that are output by the rendering unit to the display unit are correctly represented there, it is possible to carry out a check by way of a check sum comparison of the two units.
In one advantageous exemplary embodiment of the invention, a plurality of display units are provided, with which in each case a rendering unit is associated. The scene processing unit can then produce for the display units in each case different 3D output image data, in particular as a partial scene or a so-called subscene, from the whole 3D scene, and the 3D output image data can be distributed in particular subscene-wise to one or more rendering units for producing the target image data. The distribution of the 3D output image data to a plurality of rendering units also makes it advantageously possible for a single display unit to distribute specific image data to a plurality of rendering units. As a result of such a distributed rendering process, it is then also possible to use a rendering unit of a first display unit that is less occupied for rendering 3D output image data of a second display unit, the rendering unit of which is highly occupied. This advantage can further be increased by associating the target image data, for example via the underlying 3D output image data and the associated perspective camera parameters, in each case with one display unit. The rendering unit can then output the target image data to the display unit that is in each case associated therewith.
If a plurality of rendering units are provided for the rendering process, a plurality of configurations and advantages can be achieved. For one, it brings about a certain failure safety, in which, if one rendering unit fails, a different rendering unit can assume the rendering tasks thereof. In addition, there is a scalability, in which the rendering performance can be increased owing to an additional rendering unit being provided, or by an available rendering unit which is less occupied assuming additional rendering tasks for a different rendering unit that is highly occupied. Furthermore, rendering tasks can be distributed with respect to the safety relevance between different rendering units. For example, one rendering unit together with a plurality of rendering units can be reserved for the respective 3D output image data that are characterized as safety-relevant or are associated with the highest priority stage. Said rendering unit can then provide for example memory write protection for the safety-relevant data.
In particular a bidirectional interaction can exist between the rendering unit and the scene management unit to the effect that the rendering unit transmits data relating to the perspective viewing locations to the scene management unit and vice versa that the scene management unit transmits the 3D output image data to the rendering unit. The data relating to the perspective viewing locations can in turn include corresponding data of the display unit, in particular coordinate data of the display unit, or be derived therefrom. In particular if a plurality of display units are present, it can thus be ensured that overlapping or coupled representations of an image scene are effected on different display units in a perspectively correct manner.
According to another advantageous embodiment of the invention, the scene processing unit uses, for the formation of the whole 3D scene, interaction parameters that describe an interaction between 3D image elements of at least two of the client-computer systems. As a result, it is possible to compute for example coverages of 3D image elements of the client-computer systems and represent them correctly in the display unit. Optical effects such as reflections and cast shadows between 3D image elements can also be computed advantageously by the scene processing unit from the whole 3D scene and be represented later on a display unit. Here, interaction parameters that can be used are for example latency parameters, transparency parameters or reflection coefficients that are associated with 3D image elements as attributes, or illumination parameters such as luminous intensity, color and light propagation direction from a light source that is associated with a 3D image scene or is present in the 3D image scene as a 3D image element. By way of example, the image of a pedestrian symbol, that is to be represented close to a set of traffic lights, can be periodically colored slightly yellow on the display unit of a motor vehicle due to the amber light of a flashing traffic light to be represented. An interaction between 3D image elements can be dynamic if the 3D image elements are in each case recomputed or represented on the display unit within a suitable dynamic period of for example one millisecond to 50 milliseconds. It is also possible to bring about animations thereby.
So-called deferred shading can preferably take place in the rendering unit, in which the rendering process takes place in stages, wherein the process can be completed early at predetermined processing stages at which target image data that can be output are present in the display unit and wherein successively a better image quality is achieved at in each case higher process stages.
For monitoring the rendering accuracy, it is furthermore possible to provide for a rendering of a predetermined reference image in parallel with the rendering of the 3D output image data, in particular of a reference pattern and/or a much smaller mini image, with respect to the image size, compared to the 3D output image data. A decision can then be made on the basis of the rendering result of the reference image as to whether the rendering result of the 3D output image data is used for the further processing or is discarded. For the mini or reference image, in particular a rendering data set can be used that is adapted to the respective rendering unit, to the 3D image scene and/or to the parameters associated therewith. For example, a mini image that is safety-relevant can be provided in each case for a 3D image element and/or a 3D image scene. It can be adapted to the respective 3D image element in terms of image processing relevance properties such as size, position, color, complexity and so on. A plurality of mini images can be provided for one 3D image scene.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.